Learning to Rank Structured Alternatives: An Application to Incremental Processing of Natural Language

Incremental processing of the human parser is uncontroversially supported by many psycholinguistic experiments. Briefly, incrementality accounts for the intuitive fact that language is processed from left to right. An operational account of the incrementality hypothesis, called strong incrementality, is at the core of several computational models of the human parser [4]. In this version, the parser maintains a totally connected parsing structure, while scanning the input words from left to right, with no input stored in a disconnected state. The key concept in attaching the next word to the left context is the notion of connection path (CP) [3], which is the chain of syntactic nodes that must be built in order to appropriately connect the current word to its left context. The major difficulty of implementing a strongly incremental strategy is the high number of candidate CPs, which yields a hard search problem, affecting the parser accuracy and speed. This can be shortly explained as follows. Suppose we are given a new sentence = w0, . . . , wn−1 and suppose that at stage i of parsing we know the correct incremental tree Ti−1 spanning w0, . . . , wi−1. The goal of an incremental parser is then to compute the next tree Ti in order to accommodate the next word wi. However, other trees spanning w1, · · · , wi can be generated by legally attaching other CPs. The set of trees obtained by legally attaching Ti−1 to some CP is called the forest of candidates for word wi, denoted F i = {Ti,1, . . . , Ti,mi}. This set typically contains from tens to hundreds of candidates, yielding a hard search problem. Designing accurate heuristics for guiding this search problem is a difficult task [5] but, on the other hand, the availability of databases of parsed sentences (treebanks) makes the problem interesting from a machine learning perspective. We propose that the learning algorithm rely on a statistical model in charge of assigning probabilities of correctness to each candidate tree. When unseen sentences are processed, the model can be employed to rank alternative trees, sorting them by increasing probability of correctness. In this way, the parser can try first candidate trees with higher probability of correctness. Neural networks offer a very interesting way of solving the prediction problem outlined above. However, the structured nature of data requires architectures capable of dealing with rich representations. In this task, relations among syntactic entities play a crucial role and the standard attribute-value representation commonly associated with feedforward network is not sufficient. Moreover, ranking alternatives is a special form of learning. To explain this, consider the following possible formulations of the learning task: